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GAO 


United  States 

General  Accounting  Office 

Washington,  D.C.  20548 


Resovirces,  Community,  and 
Economic  Development  Division 

B-270359 

January  24, 1996 

The  Honorable  Christopher  S.  Bond 
Chairman 

The  Honorable  Dale  Bumpers 
Ranking  Minority  Member 
Committee  on  Small  Business 
United  States  Senate 

The  Honorable  Jan  Meyers 
Chairman 

The  Honorable  John  J.  LaFalce 
Ranking  Minority  Member 
Committee  on  Small  Business 
House  of  Representatives 

This  report  was  mandated  by  the  Small  Business  Research  and  Development  Enhancement  Act 
of  1992.  The  report  focuses  on  the  implementation  of  the  Small  Business  Technology  Transfer 
(STTR)  Pilot  Program,  which  was  established  by  the  act.  In  particular,  the  report  discusses 
(1)  the  quality  and  commercial  potential  of  the  site  Program’s  research  as  shown  by  technical 
evaluations  of  the  winning  proposals  in  the  first  year  of  the  sttr  Program,  (2)  how  agencies 
addressed  potential  conflicts  of  interest  resulting  firom  the  involvement  of  federally  funded 
research  and  development  centers  in  the  program,  and  (3)  ^encies’  views  on  the  effects  of  and 
need  for  the  stir  Program  in  view  of  its  close  similarity  to  the  Small  Business  Innovation 
Research  Program. 

Please  contact  me  at  (202)  512-3841  if  you  or  your  staff  have  any  questions.  Msyor  contributors 
to  this  report  are  listed  in  £g)pendix;  I. 


Victor  S.  Rezendes 
Director,  Energy  and 
Science  Issues 
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Purpose 


The  nation’s  research  institutions — ^its  universities,  federal  laboratories, 
and  nonprofit  research  institutions — ^account  for  about  a  quarter  of  all  the 
scientists  and  engineers  in  the  United  States.  In  an  effort  to  move  new 
knowledge  from  research  institutions  to  industry,  the  Congress  authorized 
the  Small  Business  Technology  Transfer  (sttr)  Pfiot  Program  for  3  years, 
beginning  in  fiscal  year  1994.  The  Congress  also  required  that  gao  report 
on  the  implementation  of  the  program.  For  this  report,  gao  reviewed 
(1)  the  quality  and  commercial  potential  of  the  srm  Program’s  research  as 
shown  by  technical  evaluations  of  the  winning  proposals  in  the  first  year 
of  the  program,  (2)  how  agencies  addressed  potential  conflicts  of  interest 
resulting  from  the  involvement  of  federally  funded  research  and 
development  centers  in  the  program,  and  (3)  agencies’  views  on  the  effects 
of  and  need  for  sttr  in  view  of  its  close  similarity  to  the  Small  Business 
Innovation  Research  (sbir)  Program. 


Background 


As  part  of  the  legislation  (P.  L.  102-564,  Oct.  28, 1992)  reauthorizing  the 
SBIR  Program,  the  Congress  established  the  sttr  Program  to  (1)  stimulate 
technological  innovation,  (2)  use  small  businesses  to  meet  federal 
research  and  development  (r&d)  needs,  (3)  foster  and  encourage  socially 
and  economically  disadvantaged  persons’  participation  in  technological 
innovation,  and  (4)  increase  the  private  sector’s  commercialization  of 
innovations  derived  from  federal  r&d. 

Five  agencies  participate  in  the  sttr  Program,  including  the  Departments 
of  Defense  (dod).  Energy  (doe),  and  Health  and  Human  Services’  National 
Institutes  of  Health  (nih);  the  National  Aeronautics  and  Space 
Administration  (nasa);  and  the  National  Science  Foundation  (nsf).  Each 
£^ency  mans^es  its  own  program,  while  the  Small  Business  Administration 
(sba)  plays  a  central  administrative  role,  issuing  policy  directives  and 
annud  reports  for  the  program. 

The  legislation  authorized  each  agency  having  an  external  r&d  budget  in 
excess  of  $1  billion  to  set  aside  not  less  than  0.05  percent  of  that  budget 
for  the  STTR  Program  in  fiscal  year  1994,  not  less  than  0. 1  percent  in  fiscal 
year  1995,  and  not  less  than  0.15  percent  in  fiscal  year  1996.  In  the  sttr 
Program’s  first  year,  the  s^encies  expended  about  $20  nulhon  and 
estimated  that  funding  will  triple  to  $60  million  in  the  third  and  last  year  of 
the  program. 

The  STIR  Program  provides  funding  for  phase  I  and  phase  II  awards.  Work 
in  phase  I  is  intended  to  determine  the  scientific,  technical,  and 
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commercial  merit  and  feasibility  of  ideas;  the  work  is  generally  not  to 
exceed  1  year.  Work  in  phase  n  further  develops  the  proposed  ideas  and  is 
generally  not  to  exceed  2  years.  The  statute  generally  limits  the  size  of 
awards  in  phases  I  and  II  to  $100,000  and  $500,000,  respectively. 

The  STTR  Program  is  closely  modeled  on  the  sbir  Program,  which  was 
established  in  1982.  The  two  programs  share  the  same  four  goals  and  other 
basic  features,  including  participation  by  many  of  the  same  agencies,  the 
use  of  a  percentage  of  the  external  budget  for  funding,  and  a  three-phase 
approach. 


However,  the  two  programs  differ  in  one  important  respect.  In  order  to  be 
eligible  for  an  stte  award,  a  small  business  must  collaborate  with  a 
nonprofit  research  institution,  such  as  a  university,  a  federally  funded 
research  and  development  center,  or  other  entity.  This  collaboration  is 
permitted  under  the  sbir  Program  but  is  not  mandatory.  This  special  sttr 
requirement,  according  to  a  1992  report,^  was  to  provide  a  more  effective 
mechanism  for  transferring  new  knowledge  firom  research  institutions  to 
industry. 

The  Congress  e3q)ressed  concern  about  potential  conflicts  of  interest 
resulting  firom  the  role  of  research  and  development  centers  in  the 
program.  For  example,  a  conflict  could  arise  if  a  center  and  a  small 
business  submitted  an  sttr  proposal  as  partners  and,  at  the  same  time,  the 
center  helped  the  agency  judge  its  own  and  other  proposals.  As  a  result, 
the  Congress  required  agencies  to  take  steps  to  avoid  these  potential 
conflicts  of  interest. 


Results  in  Brief  Federal  agencies  rated  the  quality  and  commercial  potential  of  the  Small 

Business  Technology  Transfer  Program’s  winning  proposals  favorably  in 
the  first  year  of  the  program.  While  the  quality  of  the  proposals  was  rated 
highly,  technical  ejqjerts  were  somewhat  cautious  about  the  commercial 
potential,  possibly  because  of  the  newness  of  the  program. 

Agencies  have  taken  steps  to  avoid  potential  conflicts  of  interest  that 
might  arise  because  of  the  close  connection  between  federal  ^encies  and 
research  and  development  centers.  In  addition,  the  Departments  of 
Defense  and  Energy,  which  accounted  for  almost  all  of  the  awards 
involving  such  centers,  have  taken  steps  to  prevent  the  centers  firom  using 


'H.R.  Rep.  No.  654, 102nd  Cong.,  2nd  Sess.,  pt  1  (1992).  The  report  accompanied  H.R  4400,  a 
predecessor  to  the  bill  (S.  2941)  that  was  enacted. 
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privileged  information  in  preparing  Small  Business  Technology  Transfer 
Program  proposals. 

Agency  officials  expressed  differing  views  on  the  Technology  Transfer 
Program’s  effect  on  the  Small  Business  Innovation  Research  Program  and 
other  agency  research  and  development,  although  none  of  them  indicated 
any  negative  effects  such  as  competition  between  the  two  programs  for 
quality  proposals.  The  similarity  of  the  two  programs,  however,  raises  an 
issue  about  the  need  for  the  new  program. 


Principal  Findings 


Evaluations  Were 
Favorable,  but  the  Review 
Process  Varied 


Evaluations  of  the  site  Program’s  proposals  by  technical  experts  showed  a 
generally  high  degree  of  confidence  in  the  quality  of  the  research  and 
commercial  potential.  As  one  example,  DOE  rated  the  quality  of  proposed 
research  for  aU  of  its  winning  proposals  as  high  as  that  for  those  proposals 
in  the  top  10  percent  of  research  in  the  Department  Evaluations  of  the 
commercial  potential  were  also  favorable  but  occasionally  expressed 
concern  about  the  cost  of  the  product  that  might  result  or  the  limited  size 
of  its  potential  market  Such  reservations  may  be  due  to  the  newness  of 
the  program  and  the  risk  associated  with  many  of  the  projects. 

'The  evaluation  process,  upon  which  these  findings  depended,  varied 
greatly.  In  nih,  nasa,  nsf,  and  doe,  the  selection  of  winning  proposals  relied 
on  consensus  among  several  technical  reviewers.  In  dod,  by  contrast,  a 
single  reviewer  was  firequentiy  responsible  for  the  techmcal  evaluation  of 
proposals.  Of  dod’s  105  winning  proposals,  47  received  only  one  technical 
review  that,  in  37  cases,  provided  only  hmited  analysis  to  support  the 
ratings. 


Agencies  Have  Taken  Steps 
to  Avoid  Conflicts  of 
Interest 


In  general,  the  five  agencies  have  taken  steps  to  prevent  conflicts  of 
interest  from  occurring  as  a  result  of  involvement  by  research  and 
development  centers  in  the  sttr  Program,  dod,  doe,  and  NiH  developed 
specific  policy  guidance  concerning  potential  conflicts  of  interest,  nasa 
and  NSF  relied  on  general  procedures  in  their  procurement  guidelines  to 
avoid  conflicts  of  interest,  dod  and  doe  have  taken  steps  to  prevent 
research  and  development  centers  from  gaining  an  unfair  advantage  in 
preparing  proposals,  dod  has  carefvdly  restricted  participation  by  its  ow 
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laboratories,  and  doe  has  prohibited  agency  officials  from  requesting  or 
receiving  asastance  firom  personnel  in  research  institutions  in  the 
preparation  of  research  topics.  These  two  agencies  accounted  for  aU  but  3 
of  the  32  awards  involving  research  and  development  centers  as  research 
partners  during  the  first  year  of  the  pilot  program.  Nm,  NASA,  NSF,  and  DOE 
also  made  use  of  outside  peer  review;  by  contrast,  dod  tended  to  rely  on 
technical  expertise  within  the  Department. 


Views  Differ  on  Effects  of  Agency  officials  expressed  differing  views  on  the  effect  of  the  sm 

and  Need  for  the  Program  Program  on  the  agencies’  other  r&d,  including  the  ssm  Program.  However, 

the  agencies  provided  no  evidence  at  this  early  point  in  the  program  to 
suggest  that  the  srm  Program  was  competing  for  quality  proposals  with 
the  Irmovation  Research  Program  or  reducing  the  quality  of  the  J^encies’ 
R&D  in  general.  Instead,  some  officials  noted  some  potentially  beneficial 
effects  such  as  greater  collaboration  between  small  businesses  and 
research  institutions  in  the  Innovation  Research  Program. 

The  similarity  of  the  two  programs,  however,  raises  an  issue  about  the 
need  for  the  pilot  program.  One  way  to  assess  the  need  for  the  program  is 
to  determine  its  effectiveness  in  transferring  technology  from  research 
institutions  to  the  marketplace.  Such  information  will  not  be  ascertainable 
for  at  least  several  years  because  of  the  time  needed  to  turn  an  initial 
concept  into  a  marketable  technology. 


Agency  Comments 


STTR  program  manners  and  other  officials  firom  dod,  doe,  nasa,  nih,  nsf, 
and  SBA  reviewed  a  draft  of  this  report  and  provided  oral  comments.  In 
general,  these  officials  regarded  the  report  as  accurate  and  balanced,  dod  . 
officials  said  that  even  a  single  technical  evaluation  of  a  proposal  may  be 
adequate  because  of  the  reviewer’s  level  of  expertise.  Agency  officials  also 
provided  minor  technical  corrections  that  have  been  incorporated. 
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Introduction 


The  Small  Business  Research  and  Development  Enhancement  Act  of  1992 
established  the  Small  Business  Technology  Transfer  (sttr)  Pilot  Program 
and  authorized  it  for  3  years,  beginning  in  fiscal  year  1994.  The  act  also 
reauthorized  the  Small  Business  Innovation  Research  (sbir)  Program, 
which  was  established  in  1982.  The  sttr  Program  was  modeled  closely  on 
the  SBIR  Program.  Both  of  the  programs,  for  example,  share  the  same  four 
major  goals: 

•  Stimulating  technological  innovation. 

•  Using  small  business  to  meet  federal  research  and  development  (r&d) 
needs. 

•  Fostering  and  encouraging  participation  by  minorities  and  disadvantaged 
persons  in  technological  innovation. 

•  Increasing  private-sector  commercialization  of  innovations  derived  firom 
federal  r&d. 

The  two  programs  include  many  of  the  same  agencies,  and  the  same 
agency  offices  administer  the  programs. 

In  ^ite  of  the  numerous  points  in  common,  the  two  programs  differ  in  one 
important  respect.  The  sttr  Program  requires  a  small  business  to 
collaborate  with  a  U.S.  nonprofit  research  institution,  such  as  a  university; 
a  contractor-operated,  federally  fvmded  research  and  development  center, 
or  other  entity,  in  order  to  be  eligible  for  an  award.  This  collaboration  is 
permitted  under  the  sbir  Program,  but  it  is  not  mandatory. 


A  rlTYilnictT-Qtirkn  nf  participation  in  and  funding  for  the  srrm  Program  foUowed  the 

1  ne  AUI  l  111  us  Ll  clLiUi  I  Ui  general  approach  established  for  the  sbir  Program.  Agencies  having  a 

the  STTR  Program  budget  of  more  than  $1  billion  annually  in  fis^  years  1994, 1995,  or  1996 

for  external  r&d  are  authorized  to  set  aside  a  percentage  of  this  amount  for 
the  STTR  Program.  The  act  sets  the  percentage  at  not  less  than  0.05  percent 
in  fiscal  year  1994,  not  less  than  0.1  percent  in  fiscal  year  1995,  and  not  less 
than  0.15  percent  in  fiscal  year  1996.  Five  agencies — dod,  doe,  nasa,  nih, 
and  NSF — currently  participate  in  the  program.  In  the  SBiR  Program,  by 
contrast,  the  same  five  agencies  and  six  additional  agencies  with  smaller 
budgets  for  external  r&d  were  authorized  to  set  aside  a  significantly  higher 
percentage  and  amount  of  money.  When  the  sttr  Program  reaches  its 
highest  authorized  funding  percentage  in  fiscal  year  1996,  it  will  receive 
about  $60  million,  according  to  Small  Business  Administration  (sba) 
calculations;  when  the  sbir  Program  reaches  its  maximum  funding 
percentage  in  fiscal  year  1997,  it  will  receive  about  $1  biUion. 
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As  with  SBIR,  the  legislation  required  agencies  to  solicit  proposals  for  r&d 
projects.  The  solicitation  lists  and  describes  the  topics  to  be  addressed  by 
company  proposals  and  invites  companies  to  submit  proposals  for 
consideration.  Each  agency  is  re^onsible  for  targeting  research  areas  and 
administering  its  own  sttr  funding  agreements.  Such  agreements  include  a 
contract,  grant,  or  cooperative  agreement  entered  into  between  a  federal 
agency  and  a  small  business  for  the  performance  of  experimental, 
developmental,  or  research  work  funded  in  whole  or  in  part  by  the  federal 
government. 

The  legislation  also  required  sba  to  issue  a  policy  directive  for  the  general 
conduct  of  STTR  Programs  within  federal  ^encies.  The  directive  was  to 
include  such  features  as  simplified,  standardized,  and  timely  sbbr 
solicitations;  a  simplified,  standardized  funding  process;  and  minimization 
of  the  regulatory  burden  for  small  businesses  participating  in  the  program. 
The  STTR  policy  directive  was  issued  in  August  1993  and  remains  in  effect. 
Federal  s^encies  are  also  required  to  report  key  data  to  sba. 

To  be  ehgible  for  an  sttr  award,  small  businesses  must  be 

•  independently  owned  and  operated, 

•  other  than  the  dominant  firms  in  the  field  in  which  they  are  proposing  to 
cany  out  sttr  projects, 

•  organized  for  profit, 

•  the  employer  of  500  or  fewer  employees  (including  its  affiliates),  and 

•  at  least  51  percent  owned  by  U.S.  citizens  or  lawfully  admitted  permanent 
resident  aliens. 

The  law  established  a  three-phase  structure  for  the  program.  The  first 
phase,  not  to  exceed  1  year,  is  designed  to  determine  the  scientific, 
technical,  and  commercial  merit  and  feasibility  of  a  proposed  idea  The 
second  phase,  not  to  exceed  2  years,  is  designed  to  further  develop  the 
idea  The  statute  established  $100,000  and  $500,000  as  the  general  limits 
for  phases  I  and  n  awards,  respectively.  The  third  phase  is  somewhat  more 
flexible  and  difficult  to  define.  In  general,  this  phase  is  expected  to  result 
in  commercialization  or  further  continuation  of  r&d.  No  sttr  funding  is 
provided  for  phase  IE.  Unlike  phases  I  and  n,  phase  HI  has  no  general  time 
limits.  In  addition,  phase  El  can  include  not  only  federal  non-siTR  funds 
but  private-sector  funds. 

For  selection  of  phase  I  proposal  awards,  sba’s  1993  policy  directive  stated 
that  agency  criteria  shaE  give  primary  consideration  to  scientific  and 
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technical  merit  along  with  the  potential  for  commercialization.  According 
to  the  directive,  funding  for  phase  II  shall  be  based  upon  the  results  of 
phase  I  and  the  scientific  and  technical  merit  and  commercial  potential  of 
the  phase  n  proposal. 


The  Rationale  for  the 
STTR  Program 


Given  the  similarities  between  the  sbir  and  sttr  Programs,  the  question  of 
the  rationale  for  the  sttr  Program  naturally  presents  itself.  In  the  1992 
report  cited  above,  the  Chairman  of  the  House  Committee  on  Small 
Business  provided  two  basic  answers  to  this  question  when  the  sttr 
Program  was  under  consideration.  The  report  states  that  the  program 
addresses  a  core  problem  in  U.S.  economic  competitiveness  and  that  the 
existing  sbir  Program  does  not  provide  a  direct  mechamsm  for  technology 
transfer. 

In  discussing  the  first  point,  the  report  noted  that  the  nation’s  research 
institutions — its  universities,  federal  laboratories,  and  nonprofit  research 
institutions — contain  enormous  scientific  and  technical  resources.  The 
report  also  noted  that  the  institutions  employ  one  in  four  scientists  and 
engineers  in  the  nation  and  perform  nearly  $40  billion  in  r&d  each  year,  or 
one-quarter  of  aU  R&D  conducted  in  the  United  States.  In  addition,  the 
report  stated  that  perhsqps  the  core  of  the  U.S.  competitiveness  problem  is 
the  inability  to  translate  its  worldwide  leadership  [in  science  and 
engineering]  into  technology  and  commercial  applications  that  benefit  the 
economy.  The  report  concluded  that  what  is  needed  is  an  effective, 
systematic  “technology  transfer"  mechanism  to  move  new  knowledge  from 
the  research  institution  to  industry,  where  it  can  be  exploited  for  the 
national  good. 

In  discussing  the  second  point,  the  report  noted  that  SBIR  has  turned  out  to 
be  a  remarkably  effective  mechanism  for  commercializing  ideas  in  the 
small  business  community.  However,  according  to  the  report,  SBIR  is  less 
effective  at  fostering  commercialization  of  ideas  that  originate  in 
universities,  federal  laboratories,  and  nonprofit  research  institutions,  sttr 
would  provide  a  strong  incentive  for  small  companies  and  researchers  at 
universities,  federal  laboratories,  and  nonprofit  research  institutions  to 
find  each  other  and  work  together  because  the  only  way  they  can  access 
STIR  funding  is  by  collaborating. 

Thus,  STTR  was  envisioned  primarily  as  a  technology  transfer  program,  in 
which  promising  concepts  originating  in  the  nonprofit  research 
community  would  move  toward  commercialization.  In  accomplishing  this 
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objective,  researchers  in  nonprofit  research  institutions  would  ally 
themselves  with  small  businesses  and  play  a  greater  role  in  srm  projects 
than  they  could  play  in  sbe  projects. 


Awards  Made  During 
the  First  Year  of  the 
STTR  Program 


Table  1. 1  summarizes  basic  information  about  the  phase  I  awards  made 
during  the  first  year  (fiscal  year  1994)  of  the  sttr  Program. 


Table  1.1:  An  Overview  of  STTR 
Awards,  Fiscal  Year  1994 

(Dollars  in  millions) 

Agency 

Number  of 
proposals 

Phase  1 
awards 

Amount 

Universities 

R&D 

centers 

other 

DOD 

911 

105 

$9.8 

85 

15 

5 

NIH 

283 

48 

4.7 

41 

2 

5 

NASA 

159 

21 

2.1 

17 

1 

3 

- 

NSF 

110 

11 

1.1 

11 

0 

0 

DOE 

487 

21 

2.1 

6 

14 

1 

Total 

1,950 

206 

$19.8 

160 

32 

14 

As  shown  in  the  table,  ^endes  made  206  phase  I  awards  in  the  first  year 
of  the  program,  with  dod  accounting  for  about  half  of  them.  Among  the 
small  business  affiliates,  universities  constituted  the  msyority  (78  percent) 
of  the  research  partners,  doe  was  the  only  agency  in  which  the  awardees 
formed  partnerships  with  research  and  development  centers  in  a  majority 
of  cases. 

Among  other  significant  points  about  awardees  during  this  period, 
according  to  sba  officials,  almost  two-thirds  of  the  awards  went  to 
companies  that  had  previously  received  awards  from  the  sbe  Program.  In 
addition,  the  officials  told  us  that  the  proposed  allocation  of  award  money 
between  the  small  businesses  and  the  research  institutions  showed  the 
former  receiving  about  61  percent  of  the  funding  and  the  latter  receiving 
about  39  percent. 


Objectives,  Scope, 
and  Methodology 


In  addressing  the  quality  and  commercial  potential  of  sttr  research,  we 
obtained  and  reviewed  aU  of  the  agencies’  technical  evaluations  for  206 
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proposals  that  received  phase  I  awards  on  the  basis  of  fiscal  year  1994 
solicitations.  The  technical  evaluations  were  prepared  by  experts  in 
federal  agencies  or  the  private  sector,  if  an  agency  relied  on  outside  peer 
review.  The  evaluations  played  the  primary  role  in  determining  which 
proposals  were  selected  for  funding.  The  number  of  evaluations  for  each 
award  ranged  from  one  to  as  many  as  six. 

We  reviewed  the  technical  evaluations  of  project  proposals  because  they 
were  the  only  systematic  source  of  information  on  the  quality 
commercial  potential  of  the  research.  At  this  time,  the  actual  results  of 
these  awards  cannot  be  assessed.  We  restricted  our  work  to  the  206 
awards  because  they  were  the  only  complete  set  of  awards  made  imder 
the  pilot  program  at  the  time  that  we  conducted  our  review.  The  results  of 
the  first  round  of  the  phase  II  award  process  were  not  available  for  all 
agencies  until  late  in  1995;  in  fact,  only  a  few  phase  11  awards  were  made 
during  fiscal  year  1994. 

Because  our  conclusions  about  the  quality  of  research  and  commercial 
potential  were  drawn  from  these  techmcal  evaluations  of  proposed 
projects,  we  gave  additional  attention  to  the  evaluation  process  used  by 
each  of  the  agencies.  In  our  view,  greater  confidence  could  be  placed  in 
evaluations  resulting  firom  a  thorough  and  well-documented  evaluation 
process.  In  particular,  we  noted  the  number  of  reviewers  per  proposal,  the 
number  of  proposals  per  reviewer,  and  the  level  of  analysis  resulting  firom 
the  reviews.  Our  findings  were  j^ency-specific  for  dod,  nih,  nasa,  nsf,  and 
DOE.  In  addition  to  reviewing  the  technical  evaluations,  we  obtained 
further  information  about  the  evaluation  process  firom  discussions  with 
program  officials  in  each  of  the  sttr  agencies. 

To  address  compliance  by  agencies  and  research  and  development  centers 
with  a  requirement  of  the  act  to  avoid  conflicts  of  interest,  we  obtained 
relevant  documents  firom  and  conducted  interviews  with  sttr  Program 
officials.  The  act  required  each  federal  ^ency  with  an  stte  Program  to 
develop,  in  consultation  with  the  Office  of  Federal  Procurement  Policy 
and  the  Office  of  Government  Ethics,  procedures  to  ensure  that  research 
and  development  centers  (1)  are  firee  from  organizational  conflicts  of 
interests  relative  to  the  stir  Program;  (2)  do  not  use  privileged 
mformation  gained  through  work  performed  for  an  STTR  agency  or  private 
access  to  STTR  agency  personnel  in  the  development  of  an  sttr  proposal; 
and  (3)  use  outside  peer  review,  as  appropriate. 
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To  address  the  effect,  if  any,  of  sttr  on  sbir  and  other  agency  r&d,  we 
interviewed  srm  Program  officials  at  sba  and  the  five  agencies  that  have 
STTR  Programs.As  agreed  with  the  Committees,  we  also  looked  at  the  need 
for  the  STTR  Program.  The  sttr  Program,  as  mentioned  earlier,  is  modeled 
closely  on  the  sbir  Program.  The  issue  then  arises  whether  there  is  a  need 
for  two  separate  programs.  In  addressing  this  concern,  we  reviewed  a  1992 
report  by  the  House  Committee  on  Small  Business.  The  report  helped  us  to 
identify  questions  that  are  relevant  in  determining  ttie  need  for  the  sttr  . 
Program. 

We  performed  our  audit  work  between  June  and  December  1995  in  accord 
with  generally  accepted  government  auditing  standards. 
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Agencies  generally  rated  the  quality  of  the  proposed  research  and 
commercial  potential  in  sttr  proposals  highly.  For  example,  doe  rated  the 
quality  of  research  in  all  21  of  its  winning  proposals  as  being  among  the 
top  10  percent  of  all  research  in  the  agency.  Evaluations  of  the  commercial 
potential  were  also  favorable  but  somewhat  more  cautious.  For  example, 
in  some  cases  there  were  concerns  about  the  cost  of  the  product  that 
might  result  or  the  limited  size  of  its  potential  market.  Such  reservations 
were  understandable  in  view  of  the  newmess  of  the  program  and  the 
innovation  or  risk  associated  with  many  of  the  proposed  projects.  The 
evaluation  process,  upon  w^hich  these  findings  depended,  varied  greatly.  In 
agencies  other  than  dod,  the  selection  relied  on  input  and  consensus 
among  several  (generally  three  or  four)  technical  reviewers.  In  dod,  by 
contrast,  a  sin^e  reviewer  was  frequently  responsible  for  the  technical 
evaluation. 


Evaluation  Processes 
and  Their  Results 


Table  2.1  provides  a  brief  overview  of  the  evaluation  processes  used  by  the 
five  agencies  and  the  agencies’  assessment  of  the  research  proposals. 
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Table  2.1:  An  Overview  of  the  STTR  Evaluation  Process,  Quality  of  Proposed  Research,  and  Commercial  Potential,  Fiscal 

Year  1994 

Agency 

Total 

awards 

Reviews  per 
proposal 

Single 

reviews 

Quality  of 
proposed 
research 

Commercial 

potential 

NIH  ' 

48 

4 

0 

a 

a 

DOD  (total) 

105 

1  to  6 

47 

highly  rated 

highly  rated 

Navy 

22 

1.to3 

18 

10  of  22 
perfect  scores 

10  of  22 
perfect  scores 

Army 

.21 

3 

0 

cutting  edge 
research 

highly  rated 

Air  Force/ Wright  Laboratory 

18 

1  to  3 

13 

excellent 

excellent 

Air  Force  Office  of  Scientific  Research 

13 

3  to  6 

0 

a 

a 

Ballistic  Missile  Defense  Organization 

15 

2  to  6 

0 

a 

a 

Advanced  Research  Projects  Agency 

16 

1 

16 

all  16  perfect 
scores  . 

14  perfect  scores 

NASA 

21 

3 

0 

above  average/ 
top  10  percent 

above  average/ 
top  10  percent 

NSF 

11 

3 

0 

excellent 

excellent 

DOE 

21 

3 

0 

top  10  percent 

highly  rated 

®NIH,  the  Air  Force  Office  of  Scientific  Research  (AFOSR),  and  the  Ballistic  Missile  Defense 
Organization  (BMDO)  provided  only  qualitative,  not  quantitative,  evaluations  for  this  criterion.  At 

NIH,  each  project  was  given  a  score  on  an  overall  basis  that  included  research  quality  and 
commercial  potential.  In  general,  NIH’s  proposals  were  judged  as  either  outstanding  or  excellent. 

At  AFOSR.  the  research  quality  and  commercial  potential  were  highly  rated.  At  BMDO.  it  was 
difficult  to  draw  conclusions  because  of  the  diversity  of  views  expressed  in  the  evaluations. 

XTTTTJ^  T71 _ 1 _ 

The  following  sections  summarize  (1)  the  agencies’  evaluation  processes 
and  (2)  the  quality  of  research  and  commercial  potential  identified  in  the 
technical  evaluations. 

NIH  S  Evaluation  Procoss  In  its  first  (fiscal  year  1994)  solicitation  for  sttr  proposals,  nih  identified 
and  Its  Results  seven  review  criteria,  including  the  soundness  and  technical  merit  of  the 


proposed  approach  and  commercial  potential,  which  were  used  to  select 
proposals  for  funding.  These  criteria  were  not  given  specific  weight  in  the 
evaluation  process. 

In  the  process  of  evaluation,  nih  assigned  two  reviewers  (who  provided 
written  evaluations)  and  two  additional  readers  to  each  proposal.  A  peer 
review  panel  consisting  of  10  or  more  eiiperts  was  convened.  The  four 
reviewers  and  readers  began  the  discussion  of  each  proposal  by 
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presenting  a  specific,  numerical  score  to  the  panel.  The  two  reviewers  also 
presented  their  written  statements.  The  panel  then  discussed  the  proposal 
in  further  detail  to  assist  in  the  development  of  a  final  summary  statement. 
Each  member  of  the  panel  also  provided  a  score,  which  ranged  between 
100  for  the  best  and  500  for  the  worst.  The  level  of  quality  was  defined  as 
follows: 

100  to  150:  Outstanding. 

150  to  200:  Excellent. 

200  to  250:  Very  good. 

250  to  350:  Good. 

350  to  500:  Acceptable. 

For  the  48  winning  proposals,  the  average  score  was  165  (or  “excellent”). 
Specifically,  14  proposals  were  judged  outstanding,  31  excellent,  2  very 
good,  and  only  1  good.  There  were  none  in  the  acceptable  category.  This 
overall  result  compared  favorably  with  nib’s  sbie  Program,  in  which  the 
average  score  for  aU  sbir  phase  I  projects  with  awards  in  fiscal  year  1994 
was  187.  About  94  percent  of  sttr’s  awards  were  judged  as  outstanding  or 
excellent,  which  compared  very  favorably  with  sbir’s  66  percent. 

In  contrast  to  five  practice  at  other  agencies,  no  separate  score  was 
assigned  for  the  quality  of  research  and  commercial  potential.  The  Chief  of 
the  Technology  and  Applied  Sciences  Section  said  that  the  importance  of 
individual  criteria  may  vary  firom  one  proposal  to  another.  In  his  view,  an 
overall  score  allows  for  greater  flexibility  than  the  sum  of  a  series  of 
specific  scores,  each  of  which  represents  a  fixed  percentage  of  the  total 
potential  score.  The  Chief  added,  however,  that  nib  has  organized  a 
committee  to  evaluate  its  entire  scoring  system. 

Although  the  panels  did  not  provide  scores  for  each  criterion,  they 
provided  detailed  written  analyses  of  the  proposals.  The  summaries  for 
nib’s  winning  proposals  include,  among  other  points,  a  resume  of  the 
research,  a  critique  of  strengths  and  shortcomings  in  terms  of  research, 
and  an  assessment  of  its  commercial  potential. 

nib’s  evaluations  of  the  quality  of  research  were  generally  very  favorable. 
One  evaluation,  for  example,  described  a  proposal  as  addressing  a  well 
worthwhile  problem  involving  a  new  method  of  enhancing  the  power  of 
X-ray  tubes.  The  evaluation  stated  that  the  proposed  sqiproach  was  sound, 
had  high  technical  merit,  and  was  supported  by  the  theoretical 
calculations  presented.  Statements  about  commercial  potential  were 
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similarly  favorable  but  sometimes  contained  expressions  of  concern  about 
costliness  or  other  potential  drawbacks.  In  one  instance,  involving  a  new 
approach  to  X-ray  mammography,  the  evaluation  found  that  the  proposed 
approach  likely  will  result  in  an  e3q)ensive  technology,  but  prospects  are 
good  for  a  result  that  improves  on  current  technology  and  should  eryoy  a 
very  high  market  potential.  In  another  case,  the  commercial  potential  for  a 
product  that  would  help  prevent  geriatric  wandering  was  described  as 
good  if  the  cost  of  hardware  and  system  operation  could  be  kept  low.  The 
evaluation  raised  concerns  about  the  potential  costliness  and 
sophistication  of  the  system. 


DOD  identified  four  criteria  in  making  its  phase  I  awards,  including  (1)  the 
soundness  and  technical  merit  of  the  proposed  approach  and  (2)  the 
potential  for  commercial  (government  or  private  sector)  j^plication  and 
the  benefits  ejected  to  accrue  firom  this  commercialization,  dod  specified 
that  each  of  the  four  criteria  was  worth  25  points.  Only  one  dod  s^ency, 
the  Ballistic  Missile  Defense  Organization  (bmdo),  developed  a  different  set 
of  basic  criteria  and  provided  no  quantitative  scores. 

DOD  officials  pointed  out  that  there  was  a  great  deal  of  variation  from  one 
DOD  agency  to  another  in  the  evaluation  process.  In  particular,  the  agencies 
varied  in  the  number  of  reviews  that  they  require  to  make  an  award.  In  the 
Navy,  Air  Force,  and  Advanced  Research  Projects  Agency  (arpa),  47  of  the 
winning  proposals  received  only  one  technical  review;  in  bmdo,  by 
contrast,  the  number  ranged  between  two  and  six.  Several  dod  program 
managers  noted  that  the  number  and  quality  of  the  reviews  were 
time-dependent  and  that  time  pressures  may  lead  to  fewer  or  less 
thorough  reviews.  In  fact,  of  the  47  winning  proposals  with  only  one 
review,  37  were  limited  in  the  analysis  used  to  support  the  award. 

Generally  peaking,  the  dod  evaluations  rated  the  quality  of  research  as 
favorable  for  most  of  the  winning  proposals;  as  with  nih,  the  evaluations  of 
commercial  potential  were  frequently  positive  but  accompanied  by 
caveats. 

The  Navy  Our  work  identified  a  number  of  concerns  which,  in  combination,  may 

weaken  the  reliability  of  the  Navy’s  evaluation  process.  These  concerns 
included  the  limited  number  of  reviewers  per  proposal,  the  heavy  reliance 
on  only  a  few  reviewers  for  all  of  the  proposals,  and  the  brevity  of  their 
evaluations.  The  process  itself  resulted  in  very  favorable  findings  about 
the  quality  of  research  and  the  commercial  potential. 


DOD’s  Evaluation  Process 
and  Its  Results 
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In  general,  the  Navy’s  evaluation  process  and  its  conclusions  depended 
primarily  on  the  views  of  only  one  person.  Of  the  22  awards,  18  were  made 
on  the  basis  of  a  single  review.  Two  of  the  remaining  four  received  two 
evaluations  and  the  other  two  received  three  evaluations. 

In  addition  to  the  large  preponderance  of  sin^e  reviews,  tiie  Navy  relied 
heavily  on  relatively  few  individuals  for  its  evaluations.  For  example,  one 
individual  in  the  Office  of  Naval  Research  performed  seven  of  the 
evaluations  for  winning  proposals.  The  Navy’s  program  manner  told  us 
that  this  reviewer  was  also  re^onsible  for  evaluating  13  unsuccessful 
proposals. 

As  a  third  concern,  comments  on  most  of  the  winning  proposals  were 
typically  very  brief.  Single  word  comments  such  as  “superb”  and 
“outstanding”  were  used  to  justify  scores  on  winning  proposals  that 
received  only  one  review.  In  the  case  of  one  proposal,  for  example, 
“excellent,”  “very  adequate,”  and  “top  notch”  sufficed  for  the  award  of 
nearly  perfect  scores  in  three  of  the  four  individual  categories;  a  brief 
technical  comment  Qess  than  a  sentence)  addressed  the  soundness  and 
merit  of  the  approach.  No  other  written  narrative  or  corroborating  review 
was  available  to  support  the  scores. 

We  asked  the  Navy’s  sbir/STTR  Program  manj^er  whether  the  conclusions 
and  scores  recorded  on  the  worksheets  resulted  from  previous  discussions 
■with  other  reviewers  and  represented  a  consensus  by  more  than  the  one 
reviewer  signing  the  worksheet.  The  program  manager  told  us  that  the 
reviewer  was  presenting  only  his  own  assessment;  no  consensus  was 
involved.  However,  he  added  that  generally  two  additional  officials, 
usually  he  and  one  other  person,  checked  the  individual  evaluations.  In  no 
cases  were  changes  made  as  a  result  of  these  subsequent  checks. 

In  our  view,  the  absence  of  detail  and  a  broader  consensus  may  limit  the 
reliability  of  the  results.  Other  dod  and  civilian  agencies  used  more  than 
one  reviewer  per  proposal  and  arrived  at  a  consensus.  The  Navy  program 
manager  said  that,  although  the  number  of  technical  reviews  may  remain 
limited,  he  has  already  taken  steps  to  improve  the  thoroughness  of  the 
review(s)  that  the  proposals  receive.  These  steps  include  a  greater  amount 
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of  analysis  in  the  technical  review(s)  and  greater  involvement  in  the 
subsequent  review  process  by  higher  level  scientists  within  the  Navy.^ 

We  found  that  10  of  the  22  winning  proposals  received  perfect  scores  (25 
out  of  25  points)  for  soundness  and  merit  of  approach.  The  lowest  score 
was  a  20  (whose  meaning  was  not  defined).  Generally  speaking,  the  Navy 
reviews  rated  the  commercial  potential  of  the  winning  proposals 
favorably.  Ten  of  the  proposals  also  received  perfect  scores  regarding  the 
potential  for  and  benefit  of  commercialization.  The  lowest  score  in  this 
area  was  a  15;  the  reviewer’s  main  concern  was  the  relatively  high  risk  of 
the  venture. 

The  Army  Each  of  the  Army’s  proposals  received  three  reviews.  A  core  group  of  six 

scientists  at  the  Army’s  Research  and  Technology  Integration  Office, 
which  moves  Army-funded  technology  from  universities  to  the 
marketplace,  oversaw  the  process.  Elach  of  the  Army’s  proposals  was 
assigned  to  a  staff  scientist  with  relevant  expertise  at  the  Army  Research 
Office;  this  scientist  was  responsible  for  evaluating  assigned  proposals. 
After  evaluating  a  proposal,  it  was  given  to  one  or  more  Army  scientists. 
The  evaluations  were  returned  to  the  original  reviewer,  who  would 
identify  and  resolve  any  discrepancies  and  then  assign  a  numerical  score 
to  each  of  the  four  dod  criteria  on  a  scale  of  zero  to  25. 

'The  Army  also  defined  scoring  guidelines  that  assigned  specific  meanings 
to  numerical  scores  to  keep  evaluators  consistent  with  each  other  and 
ensure  that  low  scores  would  be  assigned  if  deserved.  The  guidelines 
provide  clear  definitions  to  differentiate  the  various  levels  of  value  in  the 
proposals.  Soundness  and  technical  merit,  for  instance,  were  defined  as 
follows: 

•  Nobel-quahty  research — 25  points. 

•  Cutting-edge  research;  good  chance  of  mqjor  breakthrough — 20  points. 

•  Should  lead  to  important  product  improvements — 15  points. 

•  Should  lead  to  important  but  widely  anticipated  product 
improvements — 10  points. 

•  Should  lead  to  minor  product  improvements,  at  best — 5  points. 

•  Unsound  and/or  of  negligible  technical  merit — 0  points. 


^During  our  meeting  with  DOD  officials  in  December  1995  to  discuss  our  draft  report,  the  Navy’s  STTR 
program  manager  noted  a  key  reason  for  the  limited  number  of  technical  reviewers  used  by  the  Navy 
in  evaluating  STTR  proposals.  He  said  he  has  relied  exclusively  on  scientists  within  the  Office  of  Naval 
Research  for  technical  reviews  of  STTR  proposals.  He  has  not  included  scientists  from  other  DOD 
laboratories  because  he  hopes  that  these,  laboratories  may  eventually  be  included  as  research  partners 
in  STTR  work;  their  involvement  in  the  evaluation  process  would  create  a  conflict  of  interest  with  their 
potential  role  in  STTR-related  research. 
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Commercial  potential  was  defined  as  follows: 

•  Potential  to  create  a  mayor  industry — 25  points. 

•  Potential  to  redirect  a  mayor  industry — ^20  points. 

•  Potential  to  permit  small  business  to  grow  to  medium  size,  over  time  and 
with  expected  follow-ons — 15  points. 

•  Potential  to  launch  a  useful  new  product  or  product  line  with  a  significant 
market  (private  and/or  government) — 10  points. 

•  Potential  to  sustain  a  small  business  for  a  limited  time — 5  points. 

•  Little  commercialization  potential-r-0  points. 

The  scores  for  soundness  and  technical  merit  of  the  a4)proach  ranged 
between  17  and  23  for  the  21  winning  proposals.  Thus,  according  to  the 
Army’s  definitions,  Army  reviewers  regarded  all  of  the  winning  proposals 
as  leading  at  the  very  least  to  important  product  improvements.  Thirteen 
of  the  awards  scored  20  or  better,  meaning  that  they  were  regarded  as 
cutting-edge  research  with  a  good  chance  of  a  mjyOr  breakthrough. 
Overall,  the  average  score  of  the  winning  proposals  was  19. 

For  commercial  potential,  the  scores  ranged  between  12  and  23.  At  the 
very  least.  Army  reviewers  regarded  all  but  two  of  the  winning  proposals 
as  having  the  potential  to  permit  a  small  business  to  grow  to  medium  size. 
Thirteen  of  the  awards  scored  20  or  better,  meaning  that  they  had  the 
potential  to  redirect  a  mayor  industry.  The  award  that  scored  23  was 
midway  between  the  potential  to  redirect  a  mayor  industry  and  the 
potential  to  found  a  mayor  industry.  Overall,  the  average  score  was  about 
19. 

According  to  the  Army’s  srre  Program  manager,  srre  proposals  are 
unquestionably  higher  in  quality  than  sbir  because  researchers  from 
generally  outstanding  institutions  were  willing  to  include  their  names  in 
proposals  only  if  the  research  itself  was  excellent;  by  requiring  the 
involvement  of  research  institutions,  sttr  helped  to  improve  the  quality  of 
research.  In  addition,  the  small  size  of  the  sttr  Program  helped  to  foster 
quality;  the  proposals  were  more  “scrubbed”  in  the  selection  process.  The 
Army  made  only  21  phase  I  awards  out  of  350  proposals  in  the  first  year  of 
the  program. 


Two  laboratories  (Wright  Laboratory  and  afose)  accounted  for  all  the 
technical  evaluations  and  awards  for  31  proposals  in  the  Air  Force’s  sttr 
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Program.  Wright  Laboratory  evaluated  18  wiiming  proposals,  and  afosr 
evaluated  the  other  13. 

The  program  manager  at  the  Aeronautical  Systems  Center,  which  includes 
Wright  Laboratory  at  the  Wright-Patterson  Air  Force  Base,  said  that 
generally  only  one  person  performed  the  technical  evaluation.  There  was 
only  one  technical  review  for  each  of  13  of  the  laboratory’s  18  winning 
proposals;  two  or  three  reviewers  evaluated  the  other  winning  proposals. 
The  program  manager  added  that  each  proposal  received  subsequent 
re\iew  at  three  successively  higher  levels  by  the  branch,  division,  and 
directorate.  In  some  cases,  the  technical  reviews  were  thorough  critiques 
in  which  the  individual  reviewer  carefully  documented  the  basis  for  the 
scores  assigned  to  the  proposals.  In  other  cases,  there  was  very  little 
documentation  to  support  the  findings  of  the  evaluation. 

Reviewers  rated  the  soundness  and  technical  merit  of  aU  18  winning 
proposals  as  “excellent.”  In  addition,  all  but  1  of  the  18  were  excellent  for 
commercial  potential. 

The  Director  of  Plans  at  afosr — ^the  second  laboratory  involved  in  the  stir 
Program — said  that  the  laboratory’s  policy  is  to  have  concurrent  review 
for  each  proposal.  The  technical  mans^ers  were  e3q)ected  to  perform  an 
evaluation  and  also  send  it  to  other  reviewers.  Then,  the  managers  were 
supposed  to  distill  the  overall  findings  in  their  reviews  and  submit  the 
results  to  higher  officials  for  funding  decisions. 

AFOSR  officials  told  us  that  between  three  and  six  reviewers  participated  in 
the  laboratory’s  evaluations.  For  example,  one  of  the  technical  managers, 
who  had  signed  4  of  the  11  single  evaluation  sheets,  described  his 
jq)proach  to  the  evaluation  process.  The  manager  said  that  he  had 
personally  reviewed  23  proposals  m  his  technical  area  (packaging  for  high 
temperature  electronics),  eliminated  18,  and  focused  on  5  with  the  greatest 
merit.  Reviews  were  then  obtained  from  five  other  experts  for  each  of 
these  five  proposals.  Four  of  the  five  proposals  received  awards. 

According  to  the  technical  manager,  the  conclusions  for  each  proposal 
represented  the  consensus  of  six  concurrent  reviewers  (including 
himself). 

afosr’s  conclusions  about  the  scientific/technical  quality  and  the 
commercial  potential  of  its  research  were  very  favorable.  For  example,  tiie 
technical  manager  for  the  four  awards  involving  packaguag  for  high 
temperature  electronics  told  us  that  his  projects  had  made  vmusually  rapid 
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progress  toward  commercialization.  The  technical  manager  said  that  a 
total  of  eight  automotive  companies,  aircraft  companies,  and  geothermal 
weU-drilling  companies  expressed  strong  interest  in  the  technologies  being 
developed. 

BMDO  BMDO  devised  its  own  review  form,  which  asked  five  questions:  (1)  How 

good  is  the  science  and  engineering?  (2)  How  is  the  innovation  exciting? 

(3)  Are  the  principal  investigator,  firm,  and  research  institution  qualified? 

(4)  How  will  BMDO  benefit  after  phase  E?  (5)  How  will  phase  I  get  to  a 
phase  n  decision? 

None  of  these  criteria  referred  to  the  potential  for  commercialization, 
which  was  one  of  the  criteria  stated  in  dod’s  solicitation,  bmdo’s  site 
Program  manager  told  us  that  he  had  omitted  a  reference  to 
commercialization  because,  in  his  view,  the  reviewers  possess  technical 
expertise  but  are  not  in  a  position  to  evaluate  commercial  potential  in  the 
private  sector.  The  manager  also  said  that  he  preferred  to  emphasize 
innovation  rather  than  commercial  potential  in  selecting  phase  I  awards. 
He  added  that  his  goal  was  to  have  four  concurrent  evaluations  for  each 
proposal.  In  actual  practice  the  number  varied  firom  two  to  six. 

In  reviewing  the  technical  evaluations  of  the  15  winning  proposals,  we 
found  no  scores  assigned  to  the  individual  criteria  nor  was  an  overall 
score  provided.  The  program  manager  told  us  that  he  preferred  not  to  rely 
on  numerical  scores.  In  his  view,  numerical  scores  assigned  to  widely 
different  technologies  would  not  be  comparable.  In  addition,  the  manager 
noted  that  relatively  few  proposals  had  the  same  raters,  and  differences  in 
the  strictness  of  the  ratings  might  arise. 

Although  no  scores  were  assigned,  there  was  a  wide  range  of  qualitative 
judgments  regarding  the  merits  of  the  winning  proposals.  For  one  proposal 
with  three  reviewers,  the  fiirst  reviewer  stated  that  the  £q)proach  has  been 
known  for  many  years  and  was  a  high  school  science  fair  demonstration  in 
the  mid-1960s.  The  second  reviewer  described  the  proposal  as  intriguing 
science  and  engineering.  The  third  considered  the  proposal  as  innovative 
with  an  excellent  work  plan  to  prove  the  concept. 

Another  winning  proposal  with  six  reviewers  showed  an  equally  wide 
range  of  comments  on  the  quality  of  the  science  and  engineering.  Negative 
comments  included:  (1)  The  need  for  the  r&d  was  not  documented;  at  least 
10  laboratories  were  working  on  it.  (2)  The  proposal  did  not  provide 
enough  information  to  determine  the  quality.  Among  the  six  reviews,  the 
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only  positive  comment  was  that  both  the  science  and  engineering  were 
excellent.  The  other  three  reviewers  simply  summarized  the  approach 
without  addressing  the  question  asked  of  them:  How  good  is  the  science 
and  engineering?  The  proposal  received  four  recommendations  against 
funding  and  two  in  favor. 

In  reconciling  these  divergent  views,  the  program  manager  told  us  that  he 
made  use  of  a  panel  of  e3q)erts  and  encouraged  an  open  debate  on  the 
merits  of  a  proposal.  Each  proposal  that  received  at  least  one 
recommendation  for  funding  by  a  reviewer  was  forwarded  to  a  panel  that 
included  the  original  reviewers  and  additional  experts.  On  average,  about 
6  experts  were  included  in  each  panel,  but  the  number  ranged  from  3  to 
15.  According  to  the  program  manager,  the  panel  generally  enabled  him  to 
make  a  final  decision  on  funding  or  rejecting  each  proposal,  but  in  some 
cases  he  remained  uncertain  and  obtained  one  further  review  from  a 
recognized  expert  in  the  field. 

The  absence  of  private-sector  commercialization  as  a  criterion,  the 
absence  of  numerical  scores,  and  the  diversity  of  opinions  expressed  in 
the  evaluations  made  it  difficult  to  draw  conclusions  about  the  research 
quality  and  the  commercial  potential  of  bmdo’s  awards. 

ARPA  AKPA  used  a  20-point  scale  for  rating  its  proposals.  The  soundness  and 

technical  merit  were  worth  8  points;  the  other  three  dod  criteria  were 
worth  4  points  apiece,  aepa  required  one  technical  evaluation  with  two 
further  reviews  of  that  evaluation.  The  program  manager  for  each 
scientific  area  in  arpa’s  solicitation  was  responsible  for  providing  a  written 
technical  evaluation.  Supervisors  of  the  program  managers  reviewed  these 
evaluations  and  prioritized  the  proposals.  Then  the  sttr  Program  manager 
made  selections  mainly  on  the  basis  of  the  priority  ranking  and  program 
balance. 

Of  the  16  winning  proposals,  14  received  perfect  scores  and  2  received 
scores  of  19.  For  soundness  and  technical  merit,  aU  16  proposals  received 
the  maximum  score  of  8.  For  commercial  potential,  14  of  the  projects 
received  perfect  scores;  the  other  2  were  rated  “very  good.” 

The  narrative  portion  of  the  review  consisted  of  brief  comments  on  (1)  the 
overall  technical  evaluation  and  justification  for  selection  or  rejection  and 
(2)  the  qualifications  of  the  nonprofit  partner  and  adequacy  of 
contribution  to  the  research  project.  The  comments  provided  little 
information  on  the  quality  of  the  research  or  commercial  potential. 
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Generally,  the  comments  consisted  of  only  one  or  two  sentences.  For 
example,  one  reviewer  noted  thcit  there  is  considerable  commercial  impact 
for  the  technology  under  consideration  as  well  as  a  signij&cant  military 
payoff,  but  the  reviewer  did  not  elaborate  on  this  view. 


NASA’s  Evaluation  Process  nasa  identified  four  criteria  for  evaluating  proposals  and  assigned  them 
and  Its  Results  significantly  different  weights.  These  criteria  included  scientific/technical 

merit  (worth  20  percent  of  tiie  total  score),  the  anticipated  commercial 
applications  of  the  technologj'  (worth  40  percent),  and  two  other  criteria 
(with  a  combined  value  of  40  percent).  The  great  emphasis  placed  upon 
commercial  applications  in  the  evaluation  process  was  unique  among  the 
agencies. 

NASA  developed  a  clear  statement  of  its  selection  procedures.  According  to 
this  statement,  proposals  were  to  be  evaluated  by  a  review  team  consisting 
of  three  members — one  firom  academia,  one  from  the  private  sector,  and 
one  firom  government.  Each  reviewer  was  to  independently  review  the 
proposals.  The  NASA  technical  manager  at  the  field  center  was  required  to 
resolve  differences  and  obtain  a  consensus  on  the  merit  of  the  proposal.  In 
a  further  document  providing  “Guidelines  for  Evaluators,”  NASA  stated  that 
a  scoring  range  of  90  to  100  percent  should  be  interpreted  as  equivalent  in 
quality  to  the  top  10  percent  of  all  nasa  proposals  for  comparable  R&D.  A 
score  between  80  and  89  percent  signified  an  above  average  proposal.  Of 
the  21  winning  proposals,  11  were  considered  above  average,  and  8  were 
judged  as  being  among  the  top  10  percent  of  aU  nasa  proposals  for 
comparable  r&d.  • 


NSF’s  Evaluation  Process  nsf  identified  five  criteria  to  which  approximately  equal  consideration  was 
and  Its  Results  given.  These  criteria  included  (1)  the  scientific/en^eering  quality  and 

(2)  the  potential  for  commercial  applications  and  the  success  of  past 
commercialization  efforts. 

After  an  initial  screening  to  eliminate  any  proposals  not  responsive  to  the 
solicitation,  nsf  required  three  concurrent  reviews  of  the  remaining 
proposals.  The  reviewers  rated  each  of  the  above  criteria  on  a  five-point 
scale  (firom  poor  to  excellent).  A  “very  good”  score  was  20,  and  “excellent” 
(or  maximum)  score  was  25.  The  reviewers  then  presented  their  results  to 
a  panel,  which  developed  a  summary  for  each  proposal. 
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Of  the  11  winning  proposals,  the  quality  of  the  research  and  its  commercial 
potential  were  consistently  rated  as  favorable.  Among  the  33  reviews  of 
the  11  projects,  all  but  one  found  the  quality  to  range  between  very  good 
and  excellent.  For  commercial  potential,  the  majority  of  the  reviews  found 
the  quality  to  be  excellent.  Only  1  of  the  33  reviews  evaluated  the  quality 
as  merely  “good.” 


DOE  identified  five  criteria  with  approximately  equal  consideration  given  to 
each  of  them.  These  criteria  included  (1)  the  scientific/technical  quality  of 
the  research  and  (2)  the  anticipated  technical  and/or  economic  benefits  of 
the  proposed  research,  if  successfiil,  with  special  emphasis  on  the 
likelihood  that  the  project  aviU  attract  further  funding  for  product  or 
process  development  after  the  sttb  support  expires. 

doe’s  evaluation  process  consisted  of  two  steps.  First,  each  of  three 
reviewers  provided  written  comments  addressing  the  five  criteria  but  did 
not  assign  a  specific  score  to  the  criteria  Second,  technical  managers 
reviewed  the  three  evaluations  and  quantified  the  results. 

Of  the  21  winning  proposals,  16  received  perfect  scores.  The  other  five 
received  perfect  scores  on  four  of  the  five  criteria  The  managers 
expressed  imanimous  agreement  about  the  quality  of  the  research  in 
particular.  According  to  doe’s  definition  of  quality  as  used  in  evaluating 
STIR  proposals,  a  rating  of  “outstanding”  for  scientific/technical  merit 
indicated  that  the  proposal  was  comparable  to  the  top  10  percent  of 
projects  in  doe.  AU  21  of  the  winners  were  rated  as  outstanding. 

While  DOE  did  not  specifically  evaluate  proposals  for  commercial  potential, 
it  did  evaluate  anticipated  technical  and/or  economic  benefits.  Twenty  of 
the  21  proposals  received  outstanding  ratings;  only  1  was  rated  in  the  next 
lower  category  as  “significant”  Many  of  the  evaluations  on  which  the 
managers  based  their  conclusions  contained  favorable,  but  qualified, 
statements  about  commercial  potential.  For  example,  one  of  the  reviewers 
wrote  that,  if  the  research  is  successful,  the  technology  will  be  r£q)idly 
commercialized. 

One  reason  for  the  excellent  results,  as  noted  by  doe’s  stir  Program 
manager,  was  the  unusually  large  number  of  proposals  in  relation  to  the 
number  of  awards  available,  doe  received  487  proposals,  meaning  that  less 
than  5  percent  of  the  applicants  were  successful.  In  the  manager’s  view, 
the  program  was  excessively  competitive.  As  a  result,  he  took  two  steps  to 
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reduce  the  number  of  applicants.  First,  a  notice  was  included  in  the 
second  solicitation  to  alert  applicants  to  the  situation.  The  notice  pointed 
out  the  ratio  (1  award  to  23  proposals)  in  the  first  year  and  concluded  that 
only  those  j^pUcations  with  the  highest  scientific/technical  quality  would 
be  competitive.  Second,  because  broad  topics  tend  to  attract  more 
proposals,  the  topics  were  narrowed  in  the  new  solicitation  to  reduce  the 
number  of  proposals  and  improve  the  award/proposal  ratio  from  1  in  23  to 
about  1  in  10. 


Conclusions 


In  general,  technical  evaluations  of  srm  proposals  showed  favorable  views 
of  the  quality  of  proposed  research  and  commercial  potential.  For 
research  quality,  the  evaluations  (1)  awarded  perfect  scores  to  many 
proposals,  (2)  rated  proposals  among  the  top  10  percent  of  research  in 
certain  agencies,  (3)  described  some  proposals  as  “cutting  edge,”  and 
(4)  generally  found  the  quality  to  be  excellent.  For  commercial  potential, 
the  evaluations  arrived  at  similarly  favorable  conclusions,  although  in 
some  cases  they  were  somewhat  more  cautious  because  of  the  newness  of 
the  program  or  the  risk  associated  with  the  proposals. 

The  evaluation  processes  varied  greatly,  ranging  from  several  technical 
reviewers  to  only  one  per  proposal  and  from  detailed  critiques  to 
evaluation  sheets  that  provided  no  analysis  in  support  of  the  ratings. 
Almost  half  (47)  of  dod’s  winning  proposals  received  only  one  review;  in 
37  cases,  the  reviews  were  too  brief  to  support  the  findings.  Because  of 
their  limited  analysis  and  lack  of  a  broader  consensus,  these  instances 
tended  to  reduce  our  confidence  in  the  reliability  of  the  results. 
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The  five  federal  j^encies  with  sttr  Programs  have  taken  steps  to  avoid 
potential  problems  relating  to  conflict  of  interest  with  federally  funded 
research  and  development  centers.  In  addition,  dod  and  doe,  which 
accounted  for  29  of  the  32  awards  involving  centers  during  the  first  year  of 
the  program,  have  taken  steps  to  prevent  centers  firom  using  privileged 
information  in  preparing  stir  proposals. 


Agencies  Have  Taken 
Steps  to  Avoid 
Potential  Problems 
Resulting  From  R&D 
Center  Involvement 


The  legislation  establishing  the  sttr  Program  required  agencies  to  develop 
procedures  to  ensure  that  r&d  centers  are  firee  from  organizational 
conflicts  of  interest.  Such  conflicts  might  arise,  for  example,  if  a  center 
formed  a  partnership  with  a  company  submitting  an  stir  proposal  and 
helped  a  federal  agency  judge  the  merits  of  its  own  and  other  proposals.  A 
second  requirement  directs  each  agency  to  develop  procedures  ensuring 
that  the  centers  use  outside  peer  review,  as  appropriate.  Under  the  stir 
Program,  however,  the  agencies,  not  the  centers,  are  responsible  for 
decisions  regarding  peer  review;  accordingly,  we  have  focused  on  what 
the  agencies  have  done.  In  addition,  the  legislation  required  j^encies  to 
ensure  that  the  centers  do  not  use  privileged  information  gained  through 
work  performed  for  an  stir  agency  or  private  access  to  stir  agency 
personnel  in  the  development  of  an  sttr  proposal. 


Agencies  Have  Taken  Steps  in  general,  the  five  agencies  with  STIR  Programs  have  taken  steps  to 
to  Avoid  Conflicts  of  prevent  conflicts  of  interest  from  occurring,  dod,  doe,  and  nih  have 

Interest  specific  policies  intended  to  prevent  such  conflicts  while  nasa  and  nsf 

have  more  general  procedures  to  prevent  such  conflicts  of  interest  from 
arising. 

dod’s  Director  of  Defense  Research  and  Engineering  issued  a 
memorandum  in  mid- 1994  providing  policy  guidance  on  research  and 
development  centers’  participation  with  industry  in  stir  and  similar 
programs.  The  memorandum  concludes  that,  if  a  center  requests 
authorization  to  participate  in  such  programs,  the  mission  of  the  particular 
center  and  the  potential  for  conflict  of  interest  must  be  primary 
considerations  in  the  decision  process. 

As  a  result  of  dod’s  policy,  only  two  R&D  centers  are  currently  approved 
research  partners  for  its  stir  awardees.  The  Air  Force  had  to  rescind  some 
awards  because  the  proposed  research  partners  (certain  dod  laboratories) 
were  ineligible  to  participate.  According  to  dod’s  sttr  Program  director. 
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future  proposals  will  be  evaluated  on  a  case-by-case  basis  to  ensure  that 
conflicts  of  interest  do  not  occur. 

DOE  has  a  poUcy  addressing  conflict  of  interest  for  the  sttr  Program. 
According  to  the  policy,  doe  staff  members  should  neither  request  nor 
receive  assistance  from  personnel  in  research  institutions  (that  are  eligible 
to  participate  in  the  sttr  Program)  in  the  preparation  of  technical  topics 
for  the  STTR  solicitation.  In  addition,  no  person  affiliated  with  a  research 
institution  may  serve  as  a  reviewer  of  a  grant  application  that  names  that 
research  institution  as  a  participant.  Furthermore,  no  one  affiliated  with  a 
research  institution  may  assist  technical  managers  with  the  doe  review, 
evaluation,  and  selection  process  for  phase  I  grant  applications  in  a 
particular  scientific  area  if  that  research  institution  is  a  participant  on  any 
grant  application  submitted  to  that  scientific  area 

NIH  has  adopted  a  certification  procedure  to  avoid  conflicts  of  interest. 
nih’s  solicitation  for  proposals  requires  the  s^plicant  to  certify  that  it 
“(1)  is  free  from  organizational  conflicts  of  interests  relative  to  the  sttr 
Program,  (2)  did  not  use  privileged  information  gained  through  work 
performed  for  an  sttr  agency...and  (3)  used  outside  peer  review,  as 
appropriate,  to  evaluate  the  proposed  project  and  its  performance 
therein.  ” 

NASA  relied  on  a  peer  review  process  for  its  proposals  to  avoid  conflicts  of 
interest.  As  stated  earlier,  NASA’s  review  teams  consisted  of  three 
members — one  from  academia,  one  from  the  private  sector,  and  one  from 
the  government.  According  to  NASA’s  sttr  Program  manager,  none  of  the 
reviewers  were  connected  with  a  federally  funded  r&d  center.  The 
manager  also  mentioned  that  nasa’s  reviewers  certified  that  they  have  no 
conflict  of  interest  in  their  evaluations.  The  manager  said  that  the  only 
federally  funded  r&d  center  directly  associated  with  NASA  on  a  regular 
basis  is  the  Jet  Propulsion  Laboratory.  The  manager  said  that,  although  the 
laboratory  was  included  as  a  research  partner  in  several  proposals  by 
small  businesses,  it  was  not  involved  in  reviewing  those  proposals  or 
adiruriistering  the  program;  in  addition,  no  special  information  was 
provided  to  this  center. 

NSF  also  relied  on  peer  review  procedures  to  avoid  conflicts  of  interest.  Its 
“Proposal  and  Award  Manual”  provides  guidance  on  the  use  of  peer  review 
and  states  that  each  nsf  program  has  one  primary  method  for  peer  review 
which  represents  the  minimum  evaluation  received  by  proposals  in  that 
program.  Each  of  its  sttr  proposals  received  three  reviews,  nsf’s  sttr 
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Program  manager  told  us  that,  in  fiscal  year  1994,  none  of  nsf’s  11  awards 
involved  a  small  business  in  a  partnership  with  a  research  and 
development  center. 

In  general,  four  of  the  five  agencies  with  srm  programs  used  peer  review 
in  evaluating  STTR  proposals,  nih,  nasa,  nsf,  and  doe  relied  on  specified 
numbers  of  outside  reviewers,  nih  assigned  four  technical  reviewers  to 
each  proposal  and  provided  additional  input  through  its  peer  review 
panels,  nsf,  nasa,  and  doe  used  three,  dod  was  the  only  agency  that  relied 
mainly  on  technical  expertise  within  the  Department  rather  than  on 
outside  reviewers.  This  approach  followed  dod’s  usual  policy  in  evaluating 
proposals. 


DOD  and  DOE  Have  Taken 
Steps  to  Prevent  Research 
and  Development  Centers 
From  Using  Privileged 
Information 


DOD  and  DOE,  which  accounted  for  29  of  the  32  awards  involving  r&d 
centers  as  research  partners,  have  policies  to  prevent  centers  from  using 
privileged  information,  dod’s  policy  of  carefully  restricting  participation  by 
its  own  laboratories  helped  in  preventing  the  centers  from  using  inside 
knowledge  in  preparing  proposals.  As  mentioned  earlier,  only  two  of  dod’s 
centers  were  eligible  to  participate  in  the  program.  The  exclusion  of  the 
other  centers,  which  would  be  the  main  sources  of  privileged  information 
regarding  dod’s  research  needs,  avoided  the  potential  problem  raised  by 
this  issue. 


doe’s  policy  prohibits  agency  staff  members  from  requesting  or  receiving 
assistance  firom  personnel  in  research  institiitions  (that  are  eligible  to 
participate  in  the  sttr  Program)  in  the  preparation  of  technical  topics  for 
the  STTR  solicitation.  This  policy  is  intended  to  prevent  research 
institutions  firom  using  their  expertise  to  influence  doe’s  choice  of  sttr 
solicitation  topics.  Otherwise,  research  institutions  could  acquire  a 
significant  advantage  by  designing  topics  to  match  their  e3q)ertise  and  then 
preparing  a  proposal  in  the  same  area 
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Agency  officials  expressed  differing  views  regarding  the  effect  of  stir  on 
SBIR  and  other  agency  r&d.  Furthermore,  none  of  the  officials  indicated  any 
specifically  negative  effects  such  as  a  competition  between  the  two 
programs  for  quality  proposals.  However,  conclusive  information 
concerning  the  effect,  if  any,  of  sttr  on  sbir  and  other  agency  r&d  was  not 
available  because  of  the  program’s  newness  and  smallness.  The 
similarities  between  the  stir  and  sbir  Programs  raise  a  broader  issue 
about  the  need  for  the  sttr  Program.  The  rationale  for  the  program,  which 
points  to  certain  weaknesses  in  sbir  and  potential  strengths  in  stir, 
suggests  some  additional  questions  that  point  more  directly  toward 
evaluating  the  need  for  sttr. 


Agency  Officials 
Noted  No  Negative 
Effects  of  STTR  on 
SBIR  or  Other  Federal 
R&D 


The  legislation  establishing  the  stir  Program  required  us  to  assess  the 
effects  of  stir  on  the  sbir  Program  and  other  agency  r&d.  Our  discussions 
with  agency  officials  provided  no  evidence  to  suggest  that  stir  was 
competing  for  quality  proposals  with  sbir  or  reducing  the  quality  of  agency 
R&D  in  general.  Instead,  agency  officials  expressed  differing  views 
regarding  the  effect  of  stir  on  sbir  and  other  agency  r&d.  A  few  officials 
noted  some  potentially  beneficial  effects.  Others  said  that  sufficient  data 
were  not  yet  available  to  determine  the  effect,  if  any,  of  sttr  on  sbir  or 
other  agency  r&d. 


SBA  officials  contended  that  sttr  was  too  small  and  too  new  a  program  to 
have  any  real  effect  on  sbir  or  on  the  broader  range  of  agency  research  at 
the  present  time.  The  officials  pointed  out  that  the  program  represented 
only  0.05  percent  of  each  agency’s  external  r&d  budget  during  its  first  year 
and  that  it  was  only  1  year  old. 


nih’s  sttr  Program  manager  told  us  that  one  of  the  main  potential  effects 
of  the  program  is  that  universities  can  have  a  greater  role  than  under  sbir. 
However,  the  manager  noted  that,  in  an  sbir  survey  undertaken  by  nih 
several  years  ago,  collaboration  between  small  businesses  and  universities 
was  already  evident  in  well  over  half  of  the  sbir  projects.® 

In  contrast  to  the  view  that  stir’s  effect  was  very  limited,  the  Army’s  stir 
Program  manager  said  that  stir  was  influencing  sbir  in  a  beneficial  way.  In 
his  opinion,  stir  is  becoming  known  through  national  conferences  and 
other  channels.  As  a  result,  small  businesses  are  realizing  that  they  have 
more  credibility  and  chance  of  winning  an  award  by  collaborating  with  a 


^During  our  meeting  with  DOD  officials  to  discuss  our  draft  report,  DOD’s  STTR  program  director 
noted  that  the  STTR  Program  allows  a  research  institution  to  play  a  greater  role  in  collaborating  with  a 
small  business  than  under  the  SBIR  Program 
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university  or  other  research  institution.  The  manager  believes  that  the  sttr 
Program  has  already  led  to  more  collaboration  in  SBIR.  In  general, 
according  to  the  program  manager,  srm  is  a  promising  program  that  may 
be  as  successful  as  the  sbir  I*rogram.  The  manager  also  said  that  sttr  will 
influence  agencies  whose  research  has  traditionally  involved  the 
university  community  to  a  lesser  extent.  In  DOD,  for  example,  the  program 
manager  believes  that  sttr’s  impact  will  be  greater  than  in  certain  other 
agencies  (such  as  nih)  where  the  research  program  has  been  closely  tied  to 
the  universities. 

doe’s  sttr  Program  manager  said  that  he  has  tried  to  make  the  topics  in 
the  STTR  solicitation  somewhat  different  from  sbir.  Regarding  the  effect  of 
STTR  on  doe’s  research  program,  he  said  that  sttr  projects,  if  successful, 
will  help  to  meet  the  agency’s  r&d  needs  by  contributing  to  areas  of 
particular  interest  to  the  j^ency.  However,  the  manager  added  that  no 
conclusions  can  be  drawn  in  the  first  year  of  the  program  because  the  data 
are  too  limited. 


Three  Questions  Are 
Relevant  in 
Determining  the  Need 
for  the  STTR  Program 


The  stated  rationale  for  the  sttr  Program  and  the  mandatory  collaboration 
suggest  three  additional  questions  that  are  relevant  in  determining  the 
need  for  the  sttr  Program:  (1)  Is  the  technology  originating  primarily  in 
the  research  institution  as  envisioned  in  the  rationale  for  the  program  or  is 
it  originating  in  the  small  business?  (2)  Is  the  mandatory  collaboration 
between  the  small  business  and  the  research  institution  effective  in 
transferring  the  technology  to  the  market  place?  (3)  Can  tiie  sbir  Program 
accomplish  the  same  objective  without  the  collaboration  required  by  the 
STTR  Program? 


Is  the  Technology 
Originating  Primarily  in  the 
Research  Institution? 


The  technology  may  originate  with  the  research  institution,  the  small 
business,  or  a  combination  of  the  two.  In  the  sttr  Program,  the 
assumption  is  that  the  research  institution  will  be  the  primary  originator  of 
the  new  concept.  However,  data  to  determine  the  extent  to  which  research 
institutions  are  providing  the  core  technologies  are  not  currently  available. 
Neither  sba  nor  the  j^encies  have  collected  this  information,  doe’s 
SBIR/STTR  Program  manager  said  that  he  would  like  to  know  whether  the 
companies  or  the  research  institutions  were  drafting  the  proposals,  but 
such  information  is  not  available. 


The  relative  roles  of  the  research  institution  and  the  small  business  as  the 
source  of  the  technology  bear  directly  on  the  need  for  the  shttr  Program.  If 
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a  high  percentage  of  the  ideas  are  originating  with  small  businesses  rather 
than  with  research  institutions,  this  finding  would  raise  questions  about 
the  need  for  the  program.  On  the  other  hand,  if  a  high  percentage  of  ideas 
are  originating  with  research  institutions,  this  finding  would  suggest  that 
the  program  was  achieving  the  first  step  in  moving  ideas  from  research 
institutions  to  small  businesses. 


Is  the  Collaboration 
Effective  in  Transferring 
Ideas  From  the  Research 
Institution  to  the  Market 
Place? 


If  the  program  is  effective  in  moving  ideas  firom  research  institutions  to 
small  businesses,  then  the  next  logical  question  concerns  whether  their 
collaboration  is  effective  in  moving  them  to  the  market  place.  This 
question  can  be  approached  firom  two  directions:  (1)  Short-term  views  of 
how  weU  the  collaboration  is  working  in  general  and  (2)  long-term  data  on 
actual  commercialization. 

Information  on  how  well  the  collaboration  is  working  can  be  obtained  in 
the  near  future.  In  particular,  it  would  be  useful  to  know  how  the  small 
businesses  rated  the  contribution  made  by  their  research  partners  to  the 
research  effort.  Since  most  of  the  companies  had  not  completed  even  the 
first  phase  of  their  STTR  award  at  the  time  of  thus  report,  such  information 
was  not  available,  but  it  will  be  obtainable  in  the  next  year  or  two. 

Information  on  actual  commercial  outcomes  will  require  a  greater  amount 
of  time  before  it  can  be  obtained.  Generally,  5  to  9  years  are  needed  to  turn 
an  initial  concept  into  a  marketable  product.  Thus,  it  may  be  several  years 
before  the  commercial  effectiveness  of  the  program  can  be  evaluated. 


Can  the  SBIR  Program 
Accomplish  the  Same 
Objective  Without  STTR’s 
Mandatory  Collaboration? 


Because  one  important  difference  between  the  two  programs  is  that  sttr 
makes  a  small  business/research  institution  collaboration  mandatory,  the 
question  arises  whether  the  sbir  Program  could  accomplish  the  objective 
of  transferring  technology  firom  research  institutions  to  the  private  sector 
without  the  mandatory  collaboration.  The  rationale  for  the  sttr  Program 
tends  to  assume  that  such  collaborations  were  relatively  rare  in  the  sbir 
Program.  However,  as  noted  above,  Nffl’s  Program  manager  told  us  that,  in 
an  SBIR  survey  undertaken  by  nih  several  years  ago,  collaboration  between 
small  businesses  and  universities  was  already  evident  in  well  over  half  of 
Nffl’s  SBIR  projects.  By  contrast,  the  Army’s  program  manager  believed  that 
stir’s  impact  will  be  greater  in  the  Army  than  in  agencies  such  as  nih 
because  the  Army  sbir  Program  has  had  a  lesser  degree  of  involvement 
with  universities  and  other  research  institutions  in  the  past.  Given  the 
2q)parent  variation  firom  one  ^ency  to  another  and  the  lack  of  current 
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data,  no  definite  conclusion  can  be  drawn  at  present  concerning  the  need 
for  STTR  in  forging  new  collaborations. 

Certain  proposals  may  suggest  a  need  for  the  STTR  program.  For  example, 
one  NIH  proposal  involved  a  relationship  between  the  developer  of  a  new 
type  of  microscope  and  a  company  with  the  experience  and  capability  to 
commercialize  the  instrument  The  principal  investigator  with  the 
company  told  us  that  sttr  led  to  a  partnership  between  his  company  and 
the  research  institution  that  would  not  have  existed  otherwise.  According 
to  the  investigator,  without  sttr,  his  company  would  have  been  reluctant 
to  devote  a  lot  of  lime  to  a  technology  that  it  did  not  have  rights  to.  In  such 
a  situation,  according  to  the  investigator,  the  company  would  have 
provided  specific  components  of  the  microscope  to  the  research 
institution,  let  the  institution  develop  the  product,  and  then  pursued  a 
license  to  market  it.  Instead,  under  sttr,  the  company  developed  an 
agreement  with  the  research  institution  which  led  to  the  current 
partnership. 

However,  other  cases  may  suggest  the  opposite.  For  example,  the 
president  of  a  company  that  has  participated  in  both  the  sbir  and  sttr 
Programs  said  that,  in  his  experience,  the  sbir  Program  gave  companies  a 
greater  advantage  in  dealing  with  research  institutions  as  potential 
partners.  Under  sttr,  according  to  this  official,  the  research  institution  has 
“veto  power,”  but  under  sbir,  a  company  is  in  a  better  negotiating  position 
if  it  wants  research  to  be  done.  In  addition,  according  to  the  official,  the 
STIR  Program  will  not  be  able  to  alter  the  research-oriented  outlook  of  the 
universities  in  a  more  commercial  direction.  In  general,  the  official  said 
that  almost  all  sttr  research  could  be  accomplished  through  sbir. 
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